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DESCRIPTION 

MOVING— IMAGE SYNTHESIS DEVICE, MOVING— IMAGE SYNTHESIS METHOD, 
AND INFORMATION TERMINAL APPARATUS WITH A MOVING- IMAGE 
SYNTHESIS FUNCTION 

TECHNICAL FIELD 

The present invention relates to a moving- image 
synthesis device and a moving- image synthesis method for 
producing composite moving- image data by synthesizing (that is, 
"moving-image overlaying" or "animation overlaying") image- 
data-f or-synthesis such as graphics image data and input 
moving- image data in real time, and an information terminal 
apparatus with a moving- image synthesis function such as a 
videophone apparatus . 

BACKGROUND ART 

A conventional telephone equipped with a camera has a 
function to select one of previously stored graphics still 
images and overlay (that is, "still-image overlay") the 
selected graphics still image on a still image newly taken by 
the camera. 

r 

Further, Japanese Patent Kokai (Laid-open) Publication 
No. 2000-175166 discloses a videophone apparatus which can send 
and receive a moving- image and voice. This videophone 
apparatus has a hide function which prevents the other party 
from becoming aware of the whereabouts of a human figure 
(sender) by overlaying (that is, "still-image overlaying") a 
still image on the background of the human figure (sender) 
taken by the image pickup device to make it hard to identify 
the background or by replacing the background with a still 
image . 

However, the above-described conventional telephone 
equipped with a camera can just overlay a still image provided 
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in advance on a still image newly taken by the camera, and 
cannot send or receive a moving-image. 

Further, the above-described conventional videophone 
apparatus has a function to send and receive a moving-image, 
but when the hide function is used, a part of a sent composite 
image becomes a still image. Therefore, there is a problem 
that a part of the function to send a moving- image is wasted 
(that is, the function to send a moving- image is not put into 
full use) . 

Accordingly, an object of the present invention is to 
provide a moving- image synthesis device, a moving- image 
synthesis method, and an information terminal apparatus with a 
moving- image synthesis function, which can generate composite 
moving-image data to provide a full moving-image by 
synthesizing image-data-f or-synthesis (that is, "data for 
moving- image overlay") and the input moving- image data in real 
time . 

DISCLOSURE OF INVENTION 

A moving- image synthesis device of the present 
invention includes a synthesis processor which receives a video 
signal, which includes moving- image data and a moving- image 
control signal including display timing information of each 
frame of the moving- image data; and a storage which stores 
data-f or-synthesis , which includes a plurality of items of 
image-data-f or-synthesis and a plurality of items of control- 
data-f or-synthesis associated with the plurality of items of 
the image-data-f or-synthesis . The synthesis processor reads 
one of the plurality of items of the control-data-f or-synthesis 
from the storage at a timing based on the moving-image control 
signal; reads the image-data-f or-synthesis associated with the 
read control-data-f or-synthesis from the storage in accordance 
with the read control-data-f or-synthesis ; executes processing 
to synthesize one frame of the moving- image data and the read 
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image-data- f or-synthesis ; and repeats these processing steps to 
produce composite moving-image data* 

BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a block diagram showing a configuration of a 
moving-image synthesis device and an information terminal 
apparatus with a moving- image synthesis function , of a first 
embodiment of the present invention; 

Fig. 2 is a block diagram showing a detailed 
configuration of the data-f or-synthesis storage shown in Fig. 

l; 

Fig. 3 is a block diagram showing a detailed 
configuration of the synthesis controller shown in Fig. 1; 

Fig. 4 is a block diagram showing a detailed 
configuration of the image synthesis section shown in Fig. 1; 

Fig. 5 is a diagram showing an example of a format of 
the control-data-f or-synthesis stored in the control-data-f or- 
synthesis storage shown in Fig. 2; 

Fig. 6 is a diagram showing types of information stored 
in each control-data-f or-synthesis item (header) shown in Fig. 
5; 

Fig. 7 is a diagram showing a relationship between the 
parameters shown in Fig. 6 and the display screen; 

Fig. 8 is a diagram showing an example of a format of 
the image-data-f or-synthesis stored in the image-data-f or- 
synthesis storage shown in Fig. 2; 

Fig. 9 is a diagram showing an example of a 
relationship between the control-data-f or-synthesis and the 
image-data-f or-synthesis ; 

Fig. 10 is a diagram showing an example of display of 
the composite images in Fig. 9; 

Fig. 11 is a diagram showing another example of a 
relationship between the control-data-f or-synthesis and the 
image-data-f or-synthesis ; 
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Fig. 12 is a diagram showing an example of display of 
the composite images in Fig. 11; 

Fig. 13 is a block diagram showing a configuration of a 
moving-image synthesis device and an information terminal 
apparatus with a moving-image synthesis function, of a second 
embodiment of the present invention; and 

Fig. 14 is a block diagram showing a configuration of a 
synthesis controller in a moving-image synthesis device of a 
third embodiment of the present invention. 

BEST MODE FOR CARRYING OUT THE INVENTION 

Embodiments of the present invention will be described 
below in detail with reference to the accompanying drawings. 

First Embodiment 

Fig. 1 is a block diagram showing a configuration of a 
moving-image synthesis device 100 (a device which can implement 
a moving-image synthesis method of the present invention) and 
an information terminal apparatus 110 with a moving-image 
synthesis function, of a first embodiment of the present 
invention. The information terminal apparatus 110 is a 
cellular phone with a videophone function, for example. 

As shown in Fig. l f the information terminal apparatus 
110 includes the moving-image synthesis device 100, an image 
pickup device 1, a data-f or-synthesis input section 2, a CPU 6, 
a voice input section 7, a voice output section 8, an image 
display section 9, and a videophone processor 10 which has a 
function to communicate through a public network 11. 

Further, as shown in Fig. 1, the moving- image synthesis 
device 100 includes a data-f or-synthesis storage 3, a synthesis 
controller 4, and an image synthesis section 5. The synthesis 
controller 4 and the image synthesis section 5 compose a 
synthesis processor 20 which carries out the processing to 
synthesize (that is, "moving- image overlay") image-data-f or- 
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synthesis G2G and moving- image data GIG. The moving- image 
synthesis device 100 is configured by semiconductor integrated 
circuitry, for example. 

The image pickup device 1 is a camera, for example, and 
generates a video signal Gl including moving-image data GIG and 
moving- image control signal G1C accompanying therewith. The 
generated video signal Gl is separated into the moving- image 
data GIG, which is a portion of data actually displayed as an 
image, and the moving- image control signal G1C, which is a 
control data portion, and supplied to the moving- image 
synthesis device 100. The moving- image control signal G1C is 
supplied to the CPU 6 as well. 

The data-f or-synthesis input section 2 is connected to 
a serial communication line (not shown) conforming to a 
standard such as RS-232C and USB or connected directly to a CPU 
bus line (not shown) , and inputs data-f or-synthesis G2 to the 
moving- image synthesis device 100. The data-f or-synthesis G2 
includes image-data-f or-synthesis G2G which is a part of data 
actually displayed as an image, and control-data-f or-synthesis 
G2C which forms sequence data. 

The data-for-synthesis storage 3 is an element for 
storing the data-for-synthesis G2 supplied from the data-for- 
synthesis input section 2, and will be described later in 
further detail with reference to Fig. 2. 

The synthesis controller 4 is an element for 
controlling the synthesis processing of the moving-image data 
GIG and the image-data-f or-synthesis G2G in accordance with the 
moving-image control signal G1C contained in the video signal 
Gl and the control-data-f or-synthesis G2C contained in the 
data-for-synthesis G2 , and will be described later in further 
detail with reference to Fig. 3. 

The image synthesis section 5 is an element for 
synthesizing the moving-image data GIG input from the image 
pickup device 1 and the image-data-f or-synthesis G2G read from 
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the data-for-synthesis storage 3 in accordance with the 
control-data-for-synthesis G2C read from the data-for-synthesis 
storage 3 and a control signal from the CPU 6, which will be 
described later, and will be described later in further detail 
with reference to Fig. 5. 

The CPU (central processing unit) 6 is an arithmetic 
unit containing a processing element such as a microprocessor 
and peripheral circuitry such as a register for outputting a 
control signal predetermined for image synthesis at a timing 
given by the moving- image control signal G1C. The CPU 6 
generates a moving- image ratio control signal which is 
parameter data indicating a ratio of the moving- image in image 
synthesis, an image-f or-synthesis ratio control signal which is 
parameter data indicating a ratio of an image-f or-synthesis , 
and a control signal containing a switch control signal CPU C , 
in accordance with the input moving-image control signal G1C, 
and outputs the signals to the image synthesis section 5. In 
Fig. 1, the moving- image ratio control signal, the image-for- 
synthesis ratio control signal, a moving- image selection 
control signal, and an image-f or-synthesis selection control 
signal are collectively expressed as CPU P . 

The voice input section 7 includes a microphone or the 
like, and generates an audio signal in accordance with the 
input voice. The voice output section 8 includes a speaker or 
the like, and outputs a sound in accordance with the input 
audio signal. The image display section 9 is a liquid crystal 
display section, for example, and displays an image based on 
the composite moving- image data G3G synthesized by the image 
synthesis section 5, an image based on the moving- image data 
received by the videophone processor 10, which will be 
described later, or the like. 

The videophone processor 10 compresses the composite 
moving- image data G3G synthesized by the image synthesis 
section 5 into a videophone image signal and sends the 
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videophone image signal to the other party, together with the 
audio signal input from the voice input section 7. The 
videophone processor 10 also decompresses a signal received 
from the other party into a videophone image signal and an 
audio signal and causes an image to be displayed by the image 
display section 9 and voice to be output from the voice output 
section 8. The public network 11 is a communication network 
such as cellular phone channels, wired telephone channels, and 
the like. 

Fig. 2 is a block diagram showing a detailed 
configuration of the data-f or-synthesis storage 3 shown in Fig. 
1. 

A control-data-f or-synthesis storage 31 stores the 
control-data-f or-synthesis G2C forming sequence data, out of 
the data-f or-synthesis G2 input from an input terminal G2 in of 
the data-f or-synthesis storage 3. The control-data-f or- 
synthesis storage 31 outputs the control-data-f or-synthesis G2C 
at the address specified by the read address information input 
from the input terminal ADC in of the data-f or-synthesis storage 
3, from an output terminal G2C ou t of the data-f or-synthesis 
storage 3. 

An image-data-f or-synthesis storage 32 stores the 
image-data-f or-synthesis G2G, which is a part of data to be 
displayed as an actual image, out of the data-f or-synthesis G2 
input from the input terminal G2 in of the data-f or-synthesis 
storage 3. The image-data-f or-synthesis storage 32 outputs the 
image-data-for-synthesis G2G at the address specified by the 
read address information input from the input terminal ADG in of 
the data-f or-synthesis storage 3, from an output terminal G2G ou t 
of the data-f or-synthesis storage 3, and also outputs parameter 
data G2Gp such as synthesis conditions and a synthesis ratio 
parameter of the image-data-for-synthesis G2G, from an output 
terminal G2GP ou t of the data-f or-synthesis storage 3. 

Fig. 3 is a block diagram showing a detailed 
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configuration of the synthesis controller 4 shown in Fig. 1. 

The moving- image control signal G1C is a signal 
accompanying the moving- image data GIG input from the image 
pickup device 1, and includes a clock, a horizontal 
synchronization signal, a vertical synchronization signal, and 
a moving-image parameter. A frame reset signal generator 41 
generates a frame reset signal FRS, which indicates the 
beginning of a frame, from the moving- image control signal G1C 
input from an input terminal GlC in of the synthesis controller 
4. 

A control-data-for-synthesis read address generator 42 
generates a control data read address ADC based on the 
information given by a control-data-for-synthesis analyzer 44 
at the timing of the frame reset signal FRS output from the 
frame reset signal generator 41, and outputs the address from 
an output terminal ADC ou t of the synthesis controller 4. 

An image-data-f or-synthesis read address generator 43 
generates an image data read address ADG based on the moving- 
image control signal G1C input from the input terminal GlC in of 
the synthesis controller 4 and the information given by the 
control-data-for-synthesis analyzer 44, which will be described 
later, and outputs the address from an output terminal ADG ou t of 
the synthesis controller 4. 

The control-data-for-synthesis analyzer 44 analyzes the 
control-data-for-synthesis G2C (header) , which is read at the 
control data read address ADC by the control-data-for-synthesis 
storage 31 and input from the input terminal G2C in of the 
synthesis controller 4, and outputs analyzed information to the 
control-data-for-synthesis read address generator 42 and the 
image-data-f or-synthesis read address generator 43. 

Fig. 4 is a block diagram showing a detailed 
configuration of the image synthesis section 5 shown in Fig. 1. 

A first fader 51 attenuates the amplitude of the 
moving- image data GIG, which is output from the image pickup 
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device 1 and input through the input terminal GlG in of the 
image synthesis section 5. A second fader 52 attenuates the 
amplitude of the image-data-f or-synthesis G2G, which is output 
from the data-f or-synthesis storage 3 and input through the 
input terminal G2G± n of the image synthesis section 5. 

An adder 53 sums up the moving- image data GIG 
attenuated by the first fader 51 and the image-data-f or- 
synthesis G2G attenuated by the second fader 52, and outputs 
composite moving- image data G3G. 

A selector 54 selects the moving-image data GIG input 
from the input terminal GlG in of the image synthesis section 5, 
the image-data-f or-synthesis G2G input from the input terminal 
G2Gin of the image synthesis section 5, or the composite 
moving-image data G3G output from the adder 53 , and outputs the 
selected data from an output terminal G3G ou t of the image 
synthesis section 5. 

A switch 55 switches an image selection control signal 
to be output to the selector 54. The switch 55 changes its 
state in accordance with a switch control signal CPU C , out of 
the control signals from the CPU 6. The switch 55 outputs to 
the selector 54, either the image selection control signal CPU P 
(a moving- image selection control signal, which is parameter 
data for selecting a moving- image , and an image-f or-synthesis 
selection control signal, which is parameter data for selecting 
an image-f or-synthesis) , out of the control signals output from 
the CPU 6 and input through the input terminal CPUP in of the 
image synthesis section 5, or the moving- image selection 
control signal and the image-f or-synthesis selection control 
signal, out of the control-data-f or-synthesis G2G P , which is 
parameter data output from the data-f or-synthesis storage 3 and 
input through the input terminal G2GPi n of the image synthesis 
section 5 . 

The switch 56 selects an image ratio control signal 
output to the first fader 51 and the second fader 52. The 
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switch 56 changes its state in accordance with the switch 
control signal CPU C from the CPU 6. The switch 56 outputs to 
the first fader 51 and the second fader 52 , either the image 
ratio control signal CPU P (a moving- image ratio control signal, 
which is parameter data about a ratio of the moving- image , and 
an image-f or-synthesis ratio control signal, which is parameter 
data about a ratio of the image-f or-synthesis) , out of the 
control signals output from the CPU 6 and input through the 
input terminal CPUPi n of the image synthesis section 5, or the 
moving-image ratio control signal and the image-f or-synthesis 
ratio control signal, out of the control-data-f or-synthesis 
G2G P output from the data-f or-synthesis storage 3 and input 
through the input terminal G2GP in of the image synthesis 
section 5. 

The operation of the first embodiment will be described 
below with reference to Figs. 1 to 4 . 

The data-for-synthesis input section 2 was previously 
supplied with the data-for-synthesis G2 , either through a 
serial communication means (not shown) conforming to a standard 
such as RS-232C and USB, or directly from the CPU bus (not 
shown) . The data-for-synthesis G2 input from the data-for- 
synthesis input section 2 is supplied to the data-for-synthesis 
storage 3. The data-for-synthesis G2 is separated into the 
control-data-f or-synthesis G2C and the image-data-f or-synthesis 
G2G. Of the data-for-synthesis G2 input through the input 
terminal G2 in to the data-for-synthesis storage 3, the control- 
data-f or-synthesis G2C is stored in the control-data-f or- 
synthesis storage 31, and the image-data-f or-synthesis G2G is 
stored in the image-data-f or-synthesis storage 32. 

On the other hand, the moving- image data GIG in the 
video signal Gl output from the image pickup device 1 is input 
to the image synthesis section 5. At the same time, the 
moving- image control signal G1C, which is a control signal such 
as a clock, a horizontal synchronization signal, and a vertical 
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synchronization signal, of the video signal Gl is input to the 
synthesis controller 4 and the CPU 6. 

The moving- image control signal G1C is input through 
the input terminal GlC in of the synthesis controller 4 to the 
frame reset signal generator 41. The frame reset signal 
generator 41 catches the beginning of the vertical 
synchronization signal included in the moving- image control 
signal G1C to generate a frame reset signal FRS , for example. 
Therefore, the frame reset signal FRS becomes a signal 
representing the beginning of each frame. The frame reset 
signal FRS generated by the frame reset signal generator 41 is 
input to the control-data-f or-synthesis read address generator 
42. At a timing when the frame reset signal FRS is input, the 
control-data-for-synthesis read address generator 42 outputs 
the control data read address ADC of the data-f or-synthesis G2 , 
through the output terminal ADC ou t of the synthesis controller 4 
to the data-f or-synthesis storage 3. The control data read 
address ADC is generated on the basis of information provided 
by the control-data-f or-synthesis analyzer 44 after analyzing 
the header (control-data-f or-synthesis) of the data-f or- 
synthesis G2 read in the previous frame. 

The control data read address ADC output from the 
synthesis controller 4 is input through the input terminal 
ADCin to the control-data-f or-synthesis storage 31 in the data- 
f or-synthesis storage 3. The control-data-f or-synthesis 
storage 31 reads a stored header (control-data-f or-synthesis) 
in accordance with the input control data read address ADC, and 
outputs the header through the output terminal G2C ou t of the 
data-f or-synthesis storage 3 to the synthesis controller 4. 
The data format and the header configuration of the header 
(control-data-f or-synthesis) stored in the control-data-f or- 
synthesis storage 31 will be described later with reference to 
Fig. 5 and Fig. 6. 

The header (control-data-f or-synthesis) read by the 
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control-data-for-synthesis storage 31 in the data»for-synthesis 
storage 3 is input through the input terminal G2C in to the 
control-data-for-synthesis analyzer 44 in the synthesis 
controller 4. Of the information of analysis results of the 
control-data-for-synthesis analyzer 44, the leading address of 
the header portion of the next data-f or-synthesis (address 
obtained from "the pointer information 3116 pointing at the 
next control-data-for-synthesis (header)" in Fig. 6, which will 
be described later) is output to the control-data-for-synthesis 
read address generator 42 , and the information of the other 
parameters (information 3111 to 3115 and 3117 in Fig. 6, which 
will be described later, and the like) is output to the image- 
data-f or-synthesis read address generator 43. 

In accordance with the moving-image control signal G1C 
(a clock, horizontal and vertical synchronization signals, and 
the like) input from the image pickup device 1 through the 
input terminal GlC in of the synthesis controller 4 and the 
parameter information of analysis results of the control-data- 
for-synthesis analyzer 44, the image-data-f or-synthesis read 
address generator 43 determines whether the data-f or-synthesis 
storage 3 should obtain the image-data-f or-synthesis G2G from 
the data-f or-synthesis input section 2, so that the image 
synthesis section 5 can synthesize the moving-image data GIG 
from the image pickup device 1 and the image-data-f or-synthesis 
G2G from the data-f or-synthesis storage 3 in accordance with 
the parameters analyzed by the control-data-for-synthesis 
analyzer 44. If the image-data-f or-synthesis G2G should be 
obtained, an acquisition request is output together with the 
image-data-f or-synthesis read address ADG . 

In the data-f or-synthesis storage 3, the image-data- 
f or-synthesis storage 32 receives the image-data-f or-synthesis 
read address ADG and the data-f or-synthesis acquisition request 
from the synthesis controller 4 through the input terminal 
ADGin. The image-data-f or-synthesis G2G is read from the 
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address specified by the image-da ta-for-synthesis read address 
ADG of the image-data-f or-synthesis storage 32, and output 
through the output terminal G2G ou t to the image synthesis 
section 5. If each pixel in the image-data-f or-synthesis G2G 
has a data format including synthesis ratio parameters in the 
image synthesis section 5 (the moving-image ratio control 
signal and the image-f or-synthesis ratio control signal) , image 
selection parameters (the moving-image selection control signal 
and the image-f or-synthesis selection control signal) , and 
synthesis condition parameters and the like, these parameters 
G2GP are output through the output terminal G2GP ou t of the data- 
f or-synthesis storage 3 to the image synthesis section 5. 

As shown in Fig. 4, in the image synthesis section 5, 
the moving- image selection control signal and the image-for- 
synthesis selection control signal of the parameters (control- 
data-f or-synthesis) G2GP are input through the input terminal 
G2GP in to the input portion 55b of the switch 55, and the 
moving-image ratio control signal and the image-f or-synthesis 
ratio control signal of the parameters G2GP are input to the 
input portion 56b of the switch 56. The moving-image selection 
control signal and the image-f or-synthesis selection control 
signal from a register or the like of the CPU 6 are input 
through the input terminal CPUP in to the input portion 55a of 
the switch 55; and the moving- image ratio control signal and 
the image-f or-synthesis ratio control signal from a register or 
the like of the CPU 6 are input through the input terminal 
CPUPin to the input portion 56a of the switch 56. The CPU 6 
supplies a control signal CPU C for selecting whether a control 
signal of either the input portion 55a or the input portion 55b 
is output to the output portion 55c of the switch 55, through 
the input terminal CPUP in to the control terminal 55s of the 
switch 55. The CPU 6 also supplies a control signal CPU C for 
selecting whether a control signal of either the input portion 
56a or the input portion 56b of the switch 56 is output to the 
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output portion 56c, through the input terminal CPUP in to the 
control terminal 56s of the switch 56. 

Further, as shown in Fig. 4, the moving- image data GIG 
is supplied from the image pickup device 1, through the input 
terminal GIGin, to the image synthesis section 5. Further, the 
image-data-for-synthesis G2G is supplied from the image-data- 
for-synthesis storage 32 in the data-f or-synthesis storage 3, 
through the input terminal G2G in/ to the image synthesis 
section 5. The moving- image data GIG input from the input 
terminal GIGin of the image synthesis section 5 is supplied to 
the first fader 51 and the selector 54. Further, the image- 
data-for-synthesis G2G input from the input terminal G2Gi n of 
the image synthesis section 5 is supplied to the second fader 
52 and the selector 54. The moving- image data GIG input to the 
first fader 51 is level-adjusted (attenuated) by the first 
fader 51 and supplied to the adder 53. Similarly, the image- 
data-for-synthesis G2G input to the second fader 52 is level- 
adjusted (attenuated) by the second fader 52 and supplied to 
the adder 53 . 

The first fader 51 and the second fader 52 are 
controlled in combination, and a control signal is output from 
the switch 56. The adder 53 sums up the output of the first 
fader 51 and the output of the second fader 52, and outputs the 
results to the selector 54. The selector 54 selects and 
outputs the moving- image data GIG input from the input terminal 
GIGin, the image-data-for-synthesis G2G input from the input 
terminal G2Gi n , or the composite moving- image data G3G of the 
moving-image data GIG and the image-data-for-synthesis G2G 
output from the adder 53, in accordance with the output of the 
switch 55. The output of the selector 54 is supplied through 
the output terminal G3G ou t to the videophone processor 10. 

The videophone processor 10 compresses data from the 
image synthesis section 5 in a method defined by a standard 
such as an ITU-T (International Telecommunications Union - 
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Telecommunication Standardization Sector) recommendation H.263 
and an MPEG4 standard; carries out voice compression processing 
for the audio data input from the voice input section 7; and 
sends the compressed data to the public network 11, using a 
videophone communication procedure. Conversely, the compressed 
data from the public network 11 is separated into audio data 
and image data by the videophone processor 10, and is 
decompressed when necessary. The audio data is output to the 
voice output section 8 , and the image data is output to the 
image display section 9 . 

The format of the control-data-f or-synthesis (header) 
G2C of the control-data-f or-synthesis storage 31 in the data- 
f or-synthesis storage 3 will next be described. 

Fig. 5 is a diagram showing an example of a format of 
the control-data-f or-synthesis stored in the control-data-f or- 
synthesis storage 31 shown in Fig. 2. 

The data format shown in Fig. 5 is composed of N items 
of the control-data-f or-synthesis (headers) numbered by 0 to N— 
1 (N is a positive integer) , denoted by reference numerals 311 
to 314. The header represents the synthesis state of each 
frame when the image is varied in a given sequence. 

Fig. 6 is a diagram showing types of information stored 
in each item of the control-data-f or-synthesis (header) shown 
in Fig. 5 (for example, control-data-f or-synthesis 311) . 

The header shown in Fig. 6 is composed of parameters 
such as "a horizontal position 3111 of image-f or-synthesis" , "a 
vertical position 3112 of image- f or-synthesis" , "a horizontal 
size 3113 of image-f or-synthesis" , "a vertical size 3114 of 
image-f or-synthesis" , "pointer information 3115 pointing at the 
current image-data-f or-synthesis" , "pointer information 3116 
pointing at the next control-data-f or-synthesis (header)", and 
"a repetition count 3117 of current image-f or-synthesis . " 
Further, the structure of the control-data-f or-synthesis G2C 
(header) is not limited to this structure. The structure of 
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the control-data-f or-synthesis may store the parameters in a 
different order, for example. 

Fig. 7 is a diagram showing a relationship between the 
parameters composing the header shown in Fig. 6 and the display 
screen of a single frame of a composite moving- image . 

The scanning of one frame of the taken moving-image 91 
shown in Fig. 7 starts from the top left corner and ends in the 
bottom right corner of Fig. 7 (that is, horizontal scanning is 
carried out from left to right in Fig. 7, and vertical scanning 
is carried out from top to bottom in Fig. 7) . "The horizontal 
position 3111 of image-f or-synthesis " represents a position 

(horizontal position) where the top left-end pixel of the 
image-data-f or-synthesis G2G of the image-f or-synthesis 92 is 
superimposed (OV_POS_H in the figure) , when the position 

(horizontal position) of the top left-end pixel of the moving- 
image data GIG is used as the reference (0) . "The vertical 
position 3112 of image-f or-synthesis " represents a position 

(vertical position) where the top left-end pixel of the image- 
data-f or-synthesis G2G of the image-f or-synthesis 92 is 
superimposed (OV_POS__V in the figure) , when the position 

(vertical position) of the top left-end pixel of the moving- 
image data GIG is used as the reference (0) . "The horizontal 
size 3113 of image-f or-synthesis " represents the horizontal 
size (OV__HSIZE in the figure) of the image-data-f or-synthesis 
G2G . "The vertical size 3114 of image-f or-synthesis" 
represents the vertical size (OV_VSIZE in the figure) of the 
image-data-f or-synthesis G2G . 

"The pointer information 3115 pointing at the current . 
image-data-f or-synthesis" of the header represents the image- 
data-f or-synthesis G2G corresponding to the header. Based on 
the pointer information 3115 pointing at the current image- 
data-f or-synthesis , the address where the first data of the 
image-data-f or-synthesis G2G is stored in the data-for- 
synthesis storage 3 can be obtained. Further, "the pointer 
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information 3116 pointing at the next control-data-for- 
synthesis (header) * of the header points at the next header 
data. Based on the pointer information 3116 pointing at the 
next control-data-for-synthesis (header) , the address where the 
first data of the next header data is stored can be obtained. 
The parameter for determining the header data to be referenced 
next (indicating the image-f or-synthesis to be referenced next, 
based on the parameter of "the leading address 3115 of the 
current image-data-for-synthesis") is important when a 
plurality of items of the header data form a sequence. 

"The repetition count 3117 of current image-for- 
synthesis" of the header indicates a repetition count of the 
image-data-for-synthesis G2G currently being used for synthesis 
processing of the image-data-for-synthesis G2G and the moving- 
image data GIG (number of repetition) . 

The format of the image data stored in the image-data- 
for-synthesis storage 32 of the data-f or-synthesis storage 3 
will *next be described. Fig. 8 is a diagram showing an example 
of a format of the image-data-for-synthesis G2G stored in the 
image-data-for-synthesis storage 32 . 

The data format shown in Fig. 8 is composed of M items 
of the image-data-for-synthesis numbered by 0 to M-l (M is a 
positive integer), denoted by reference numerals 321 to 324. 
The image-data-for-synthesis G2G of M frames are stored here. 
The data of each frame of the image-data-for-synthesis G2G do 
not need to be of the full size of one frame. Just the image 
data corresponding to the size of "the horizontal size 3113 of 
image-f or-synthesis" multiplied by "the vertical size 3114 of 
image-f or-synthesis" should be provided as described above. A 
composite image provided by "the horizontal size 3113 of image- 
f or-synthesis" and "the vertical size 3114 of image-for- 
synthesis", which are control parameters related to the size of 
the composite image, and "the horizontal position 3111 of 
image-f or-synthesis" and "the vertical position 3112 of image- 
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f or-synthesis" , which are control parameters related to the 
position of the composite image becomes like one denoted by the 
reference numeral 92 shown in Fig. 7. 

Fig. 9 is a diagram showing an example of a 
relationship between the control-data-f or-synthesis G2C and the 
image-data-f or-synthesis G2G , and Fig. 10 is a diagram showing 
an example of display of the composite images formed in Fig. 9. 
Further, Fig. 11 is a diagram showing ' another example of a 
relationship between the control-data-f or-synthesis G2C and the 
image-data-f or-synthesis G2G , and Fig. 12 is a diagram showing 
an example of display of the composite images formed in Fig. 11. 

Fig. 9 and Fig. 10 show a case where "the repetition 
count 3117 of current image-f or-synthesis " of each header is 
zero (that is, a case of displaying the image-data-f or- 
synthesis item G2G in one frame and displaying the next image- 
data-f or-synthesis item G2G in the next frame) . Fig. 11 and 
Fig. 12 show a case where "the repetition count 3117 of current 
image-for-synthesis" of the second header G2C 2 from the top is 
one (that is, a case of displaying an image-data-f or-synthesis 
G2G in two consecutive frames) . 

The example shown in Fig. 9 and Fig. 10 will now be 
described. The sequence of this example is specified as 
follows: 

- The header G2Ci is read from the control-data-f or-synthesis 
storage 31, and consequently, the image-data-f or-synthesis G2Gi 
is read from the image-data-f or-synthesis storage 32 ; 

- The header G2C 2 is read from the control-data-f or-synthesis 
storage 31, and consequently , the image-data-f or-synthesis G2G3 
is read from the image-data-f or-synthesis storage 32; 

- The header G2C 3 is read from the control-data-f or-synthesis 
storage 31, and consequently, the image-data-f or-synthesis G2Gi 
is read from the image-data-f or-synthesis storage 32; 

- The similar processing is repeated; and 

- The header G2C N is read from the control-data-f or-synthesis 
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storage 31, and consequently, the image-data-f or-syn thesis G2G 2 
is read from the image-data-f or-synthesis storage 32. 
This example is the case where "the repetition count 3117 of 
current image- f or-synthesis" of each header is zero. 

In Fig. 9, when t = ti, the header G2Ci is read from the 
control-data-for-synthesis storage 31 before the moving- image 
data GIGi is input from the image pickup device 1. In 
accordance with the contents of the header G2Ci, the image- 
data-f or-synthesis G2Gi is read from the image-da ta-for- 
synthesis storage 32 at a timing based on the input timing of 
the moving- image data GIGi, and the image synthesis section 5 
produces the composite image G3Gi and outputs the composite 
image G3Gi to the videophone processor 10. To be more precise, 
switching for each pixel is carried out in accordance with a 
control signal to the selector 54 in the image synthesis 
section 5 so that the moving- image data GIGi, the image-data- 
f or-synthesis G2Gi, or the composite image G3Gi formed from the 
moving-image data GIGi and the image-data-f or-synthesis G2Gi is 
selectively output to the videophone processor 10. 

When t = t2, the header G2C 2 is read from the control- 
data-f or-synthesis storage 31 before the moving-image data GIG2 
is input from the image pickup device 1. The reason why the 
header G2C 2 is read is because "the pointer information 3116 
pointing at the next control-data-f or-synthesis (header) " in 
the header G2Ci points at the header G2C 2 . Accordingly, there 
is no need to read the headers G2C in the order in which the 
headers G2C were stored in the control-data-f or-synthesis 
storage 31. Further, there is no need to store the headers G2C 
in the order of the sequence. The image-data-f or-synthesis 
G2G3 is read from the image-data-f or-synthesis storage 32 at a 
timing based on the input timing of the moving- image data G1G 2 , 
in accordance with the contents of the header G2C 2 , and then 
the image synthesis section 5 produces the composite image G3G 2 
and outputs the composite image G3G 2 to the videophone 
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processor 10 . 

When t = t 3 , the header G2C 3 is read from the control- 
data-for-synthesis storage 31 before the moving-image data G1G 3 
is input from the image pickup device 1. The reason why the 
header G2C 3 is read is because "the pointer information 3116 
pointing at the next control-data-f or-synthesis (header)" in 
the header G2C 2 points at the header G2C 3 . In accordance with 
the contents of the header G2C 3 , the image-data-f or-synthesis 
G2Gi is read from the image-data-f or-synthesis storage 32, at a 
timing based on the input timing of the moving- image data G1G 3 , 
and then the image synthesis section 5 produces the composite 
image G3G 3 and outputs the composite image G3G 3 to the 
videophone processor 10. The header G2C 3 references the image- 
data-f or-synthesis G2Gi, as in the case of the header G2Ci. If 
"the horizontal size 3113 of image-f or-synthesis" and "the 
vertical size 3114 of image-f or-synthesis" are the same, the 
same image-data-f or-synthesis G2G may be referenced. 

When t = tii, the header G2C N is read from the control- 
data-f or-synthesis storage 31 before the moving-image data G1G N 
is input from the image pickup device 1. The reason why the 
header G2C N is read is because "the pointer information 3116 
pointing at the next control-data-f or-synthesis (header) " in 
the previous header G2C points at the header G2C N . The 
previous header G2C is not necessarily the (N-l)-th header G2C 
stored in the control-data-f or-synthesis storage 31. The 
image-data-f or-synthesis G2G2 is read from the image-data-f or- 
synthesis storage 32 in accordance with the contents of the 
header G2C N at a timing based on the input timing of the 
moving-image data G1G N , and the image synthesis section 5 
produces the composite image G3Gn and outputs the composite 
image G3Gn to the videophone processor 10. 

When t = t N +i, the header G2Ci is read from the control- 
data-f or-synthesis storage 31 before the moving- image data 
G1Gn+i is input from the image pickup device 1. The reason why 



20 



544016WOT 



the header G2Ci is read is because "the pointer information 
3116 pointing at the next control-data-for-synthesis (header)" 
in the header G2C N points at the header G2Ci. Even if the 
control-data-f or-synthesis storage 31 stores N headers G2C , all 
the N headers do not have to be used. If "the pointer 
information 3116 pointing at the next control-data-for- 
synthesis (header)" of the (N-l)-th header G2C is set to point 
at the header G2Ci, the N-th header G2C will never be read. 
The image-data-f or-synthesis G2Gi is read from the image-data- 
f or-synthesis storage 32 at a timing based on the input timing 
of the moving- image data GlG N +i, in accordance with the contents 
of the header G2Ci, and the image synthesis section 5 produces 
the composite image G3G N +i and outputs the composite image G3G N +i 
to the videophone processor 10. 

In accordance with the sequence formed by the headers 
from G2Ci to G2C N , the image-data-f or-synthesis G2G input to 
the image synthesis section 5 are further updated, and the 
composite image data G3G are produced from the moving- image 
data GIG and the image-data-f or-synthesis G2G. 

Next, an example shown in Fig. 11 and Fig. 12 will be 
described. The sequence of this example is specified as 
follows : 

- The header G2Ci is read from the control-da ta-f or-synthesis 
storage 31, and consequently, the image-data-f or-synthesis G2Gi 
is read from the image-data-f or-synthesis storage 32; 

- The header G2C 2 is read from the control-data-f or-synthesis 
storage 31, and consequently, the image-data-f or-synthesis G2G3 
is read from the image-data-f or-synthesis storage 32: 

- The header G2C3 is read from the control-data-f or-synthesis 
storage 31, and consequently, the image-data-f or-synthesis G2Gi 
is read from the image-data-f or-synthesis storage 32; 

- The similar processing is repeated; and 

- The header G2C N is read from the control-data-f or-synthesis 
storage 31, and consequently, the image-data-f or-synthesis G2G2 



21 



544016WOT 



is read from the image-data- f or-synthesis storage 32. 

In this example, "the repetition count 3117 of current image- 

f or-synthesis" of the header G2C 2 is one. 

When t = ti, the header G2Ci is read from the control- 
data-f or-synthesis storage 31 before the moving-image data GIGi 
is input from the image pickup device 1 . In accordance with 
the contents of the header G2Ci, the image-data-f or-synthesis 
G2Gi is read from the image-data-f or-synthesis storage 32 at a 
timing based on the input timing of the moving- image data GIGi, 
and the image synthesis section 5 produces the composite image 
G3Gi and outputs the composite image G3Gi to the videophone 
processor 10. To be more precise, switching for each pixel is 
carried out by a control signal to the selector 54 in the 
synthesis section 5, and the moving-image data GIGi, the image- 
data-f or-synthesis G2Gi, or the composite image G3Gi of the 
moving-image data GIGi and the image-data-f or-synthesis G2Gi is 
selectively output to the videophone processor 10 . 

When t = t2, the header G2C2 is read from the control- 
da ta-f or-synthesis storage 31 before the moving-image data GIG2 
is input from the image pickup device 1. The reason why the 
header G2C 2 is read is just because "the pointer information 
3116 pointing at the next control-data-f or-synthesis (header) " 
in the header G2Ci points at the header G2C 2 . Accordingly, 
there is no need to read the headers G2C in the order in which 
the headers were stored in the control-data-f or-synthesis 
storage 31. Further, there is no need to store the headers G2C 
in the order of the sequence. In accordance with the contents 
of the header G2C 2 , the image-data-f or-synthesis G2G 3 is read 
from the image-data-f or-synthesis storage 32 at a timing based 
on the input timing of the moving- image data G1G 2 , and the 
image synthesis section 5 produces the composite image G3G 2 and 
outputs the composite image G3G 2 to the videophone processor 10. 

When t = t3, the header G2C 2 is read from the control- 
data-f or-synthesis storage 31 before the moving-image data GIG3 
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is input from the image pickup device 1. The reason why the 
header G2C 2 is read is because "the repetition count 3117 of 
current image-f or-synthesis" in the header G2C 2 read when t = 
t 2 is set to one. In another configuration, if "the repetition 
count 3117 of current image-f or-synthesis" in the header G2C 2 
read when t = t 2 is set to one, the header G2C is not read from 
the control-da ta-f or-synthesis storage 31 at the next timing 
(when t = t3) , and the contents of the header G2C when t = t 2 
may be continued to be held. In accordance with the contents 
of the header G2C 2 , the image-data-f or-synthesis G2G3 is read 
from the image-data-f or-synthesis storage 32 at a timing based 
on the input timing of the moving- image data GIG3, and the 
image synthesis section 5 produces and outputs the composite 
image G3G3 to the videophone processor 10. 

When t = t 4 , the header G2C 3 is read from the control- 
data-f or-synthesis storage 31 before the moving- image data GIG4 
is input from the image pickup device 1. The reason why the 
header G2C3 is read is because "the pointer information 3116 
pointing at the next control-data-f or-synthesis (header) " in 
the header G2C 2 points at the header G2C3. In accordance with 
the contents of the header G2C3, the image-data-f or-synthesis 
G2Gi is read from the image-data-f or-synthesis storage 32 at a 
timing based on the input timing of the moving- image data GIG4, 
and the image synthesis section 5 produces the composite image 
G3G4 and outputs the composite image G3G4 to the videophone 
processor 10. The header G2C 3 references the image-data-f or- 
synthesis G2Gi as in the case of the header G2Ci . If "the 
horizontal size 3113 of image-f or-synthesis " and "the vertical 
size 3114 of image-f or-synthesis" are the same, the same image- 
data-f or-synthesis G2G may be referenced. 

When t = t N+ i, the header G2C N is read from the control- 
data-f or-synthesis storage 31 before the moving- image data 
GlG N+ i is input from the image pickup device 1 . The reason why 
the header G2C N is read is because "the pointer information 
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3116 pointing at the next control-data-for-synthesis (header) - 
in the previous header G2C points at the header G2C N . There is 
no need of the previous header G2C being the (N-l)-th header 
G2C stored in the control-data-f or-synthesis storage 31. In 
accordance with the contents of the header G2C N , the image- 
da ta-f or-synthesis G2G2 is read from the image-data-f or- 
synthesis storage 32 at a timing based on the input timing of 
the moving- image data GlG N +i, and the image synthesis section 5 
produces the composite image G3Gn+i and outputs the composite 
image G3Gn+i to the videophone processor 10. 

When t = tN+2r "the header G2Ci is read from the control- 
data-f or-synthesis storage 31 before the moving-image data 
G1Gn+2 is input from the image pickup device 1 . The reason why 
the header G2Ci is read is because "the pointer information 
3116 pointing at the next control-data-f or-synthesis (header) " 
in the header G2C N points at the header G2Ci. There is no need 
to use all of the N headers G2C even if the N headers are 
stored in the control-data-f or-synthesis storage 31. If "the 
pointer information 3116 pointing at the next control-data-f or- 
synthesis (header)" of the (N-l)-th header G2C points at the 
header G2Ci, the N-th header G2C will not be read. In 
accordance with the contents of the header G2Ci, the image- 
data-f or-synthesis G2Gi is read from the image-data-f or- 
synthesis storage 32 at a timing based on the input timing of 
the moving- image data GlG N +2, and the image synthesis section 5 
produces the composite image G3G N+2 and outputs the composite 
image G3G N +2 to the videophone processor 10. 

The number of headers G2C stored in the control-data- 
f or-synthesis storage 31 and the number of items of image-data- 
f or-synthesis G2G stored in the image-data-f or-synthesis 
storage 32 (the number of frames) will now be described. . 

The number of headers G2C and the number of items of 
image-data-f or-synthesis G2G are not subject to important 
constraints. However, from the point of view of the memory 
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size, it is practically valuable that the number of the headers 
G2C is not less than the number of items of the image-data-f or- 
synthesis G2G. In that case, by specifying the same image- 
data-f or-synthesis G2G for a still (or slow-moving) frame 
(scene) , the memory size can be reduced and a long sequence can 
be specified. If the same image-data-f or-synthesis G2G can be 
specified for an A frame and a B frame (the tenth frame and the 
twentieth frame, for example) in the same sequence (if the 
tenth frame and the twentieth frame can use the same image- 
data-f or-synthesis ) , the memory size can be reduced further and 
a further long sequence can be specified. Of course, all the 
stored headers G2C and items of image-data-f or-synthesis G2G 
need not be used. A necessary header G2C and a necessary 
image-data-f or-synthesis G2G can be used in a desired order to 
form a sequence. 

Further, "the repetition count 3117 of current image- 
f or-synthesis" can be separately specified for each frame. 
This is effective when the speed of motion should be varied in 
a sequence by using a small number of items of the image-data- 
f or-synthesis G2G. This can also reduce the memory size. 

Because the image synthesis section 5 forms a composite 
image by reading the image-data-f or-synthesis G2G from the 
image-data-f or-synthesis storage 32 in synchronization with the 
input of the moving-image data GIG from the image pickup device 
1, the composite image can be produced without interruption 
within a frame. 

If the frame rate of the moving-image data GIG input 
from the image pickup device 1 is high (if a large number of 
frames are input in one second) , time-varying of the image- 
data-f or-synthesis G2G can be appropriately adjusted by setting 
"the repetition count 3117 of current image-f or-synthesis " 
(setting the repetition count to high, for example). 

As has been described above, the moving- image synthesis 
device 100 of the first embodiment can automatically synthesize 
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the image-data- f or-synthesis G2G forming a frame of a moving- 
image and the moving-image data GIG in synchronization with the 
moving- image data GIG from the image pickup device 1 by setting 
a synthesis sequence in the control-data-f or-synthesis storage 
31 in the data-f or-synthesis storage 3, so that a moving 
composite image can be formed without putting a load on the CPU 
such as when the CPU executes software. 

Further, the images are synthesized in synchronization 
with frames, so that the frames will not be output midway 
through synthesis processing. Therefore, good composite 
moving- images can be obtained. 

Furthermore, by causing the headers G2C and the image- 
data-f or-synthesis G2G forming a sequence to be related with 
each other, and a plurality of sequences can share the image- 
da ta-f or-synthesis G2G, so that the amount of image-data-f or- 
synthesis G2G to be provided can be suppressed. 

Moreover, a repetition count can be specified 
separately for each frame, so that a moving- image overlay with 
varying speed can be carried out. 

In addition, an image formed by overlaying an animated 
overlay image on a taken image can be sent through a videophone 
channel, and many other applications can be found in 
information terminal apparatuses, especially in cellular phones 
and the like. 

Further, the moving- image synthesis device 100 of the 
first embodiment holds the size of the image-data-f or-synthesis 
G2G and the image synthesis position of the image-data-f or- 
synthesis G2G corresponding to the moving-image data GIG output 
from the image pickup device 1 and other moving- image input 
stage as the data of the header G2C, so that the image-data- 
f or-synthesis G2G and the moving-image data output from the 
moving-image input stage can be synthesized in a given position 
and in a given size. 

Furthermore, the moving-image synthesis device 100 of 
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the first embodiment holds a pointer pointing at the address 
where the image-data-f or-synthesis G2G of the current frame is 
stored and a pointer pointing at the data of the header G2C of 
the next frame, as the data of the header G2C , so that the 
items of image-data-f or-synthesis G2G and the headers G2C can 
be stored in given addresses and in given orders , in the data- 
f or-synthesis storage . 

Moreover, the moving- image synthesis device 100 of the 
first embodiment holds the repetition count of the image-data- 
f or-synthesis G2G of the current frame as the data of the 
header G2C, so that the sequence formed with the image-data- 
f or-synthesis G2G can be specified to an optimum speed if the 
moving-image data GIG is input from the moving- image input 
stage at a high frame rate. 

In addition, the moving-image synthesis device 100 of 
the first embodiment can set the number of frames of the image- 
data-f or-synthesis G2G to be stored in the data-f or-synthesis 
storage to the same number as or a lower number than the number 
of frames of the data of the header G2C, so that a long 
sequence of moving-image can be synthesized with a small amount 
of image-data-f or-synthesis G2G, and the memory can be 
efficiently used in relation to the length of the sequence. 

Second Embodiment 

Fig. 13 is a block diagram showing a configuration of a 
moving-image synthesis device 2 00 (a device which can implement 
the moving-image synthesis method of the present invention) and 
an information terminal apparatus 210 with a moving- image 
synthesis function, including the moving- image synthesis device 
200, of a second embodiment of the present invention. Those 
structures of the second embodiment in Fig. 13 that are 
identical to or correspond to structures of the first 
embodiment are assigned identical symbols. 

The information terminal apparatus 210 of the second 
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embodiment shown in Fig. 13 is a composite moving-image display- 
apparatus which can synthesize an image-f or-synthesis and an 
input video signal in real time and can display a composite 
moving- image . 

The information terminal apparatus 210 of the second 
embodiment shown in Fig. 13 differs from the apparatus of the 
first embodiment shown in Fig. 1 in the following points: the 
image pickup device 1 is replaced by a video signal input 
section 12; the voice input section 7, the voice output section 
8, and the videophone processor 10 are eliminated; and the 
image display section 9 is connected directly to the image 
synthesis section 5. 

The video signal input section 12 is connected to a 
video input terminal (not shown) or is connected directly to a 
CPU bus (not shown) or the like, and receives the input of a 
digital video signal and a decode signal of MPEG4 or the like 
(collectively referred to as a digital video signal) . 

In the second embodiment, a digital video signal input 
to the video signal input section 12 is separated into moving- 
image data GIG and a moving- image control signal G1C. The 
moving-image data GIG is input to the image synthesis section 5, 
and the moving-image control signal G1C is input to the 
synthesis controller 4 and the CPU 6. The image synthesis 
section 5 and the synthesis controller 4 perform the same 
processing as in the first embodiment, and form the composite 
moving-image data G3G by synthesizing the moving-image data GIG 
and the image-data-f or-synthesis G2G. 

The moving-image synthesis device 2 00 or the 
information terminal apparatus 210 having a moving-image 
synthesis function of the second embodiment can automatically 
synthesize the image-data-f or-synthesis G2G forming a frame of 
a moving- image and the moving- image data GIG, which is a 
digital video signal from the video signal input section 12, in 
synchronization with the moving- image data GIG by setting a 
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synthesis sequence in the control-data-for-synthesis storage 31 
in the data-f or-synthesis storage 3, so that a moving composite 
image can be formed without putting a load on the CPU such as 
when the CPU executes software. 

Further, the images are synthesized in synchronization 
with frames, so that the frames will not be output midway 
through synthesis processing. Therefore, good composite 
moving- images can be obtained. 

Furthermore, by causing the headers G2C and the image- 
data-f or-synthesis G2G forming a sequence to be related with 
each other, and a plurality of sequences can share the image- 
data-f or-synthesis G2G, so that the amount of image-data-f or- 
synthesis G2G to be provided can be suppressed. 

Moreover, a repetition count can be specified 
separately for each frame, so that a moving-image overlay with 
varying speed can be carried out. 

Except for the above described respects, the second 
embodiment is the same as the first embodiment. 

Third Embodiment 

Fig. 14 is a block diagram of a synthesis controller 4a 
in a moving-image synthesis device (a device which can 
implement the moving-image synthesis method of the present 
invention) of a third embodiment of the present invention. 

The moving- image synthesis device of the third 
embodiment differs from the moving- image synthesis device of 
the first embodiment shown in Fig. 1 in that the synthesis 
controller 4 of the first embodiment is replaced by the 
synthesis controller 4a shown in Fig. 14. 

The synthesis controller 4a of the third embodiment 
shown in Fig. 14 differs from the synthesis controller 4 of the 
first embodiment shown in Fig. 3 in that the synthesis 
controller 4a has a frame rate detector 45, and the control- 
data-f or-synthesis analyzer 44 of the first embodiment is 
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replaced by the control-data-for-synthesis analyzer 46 of the 
third embodiment. The control-data-f or-synthesis analyzer 46 
receives a frame rate FRT output from the frame rate detector 
45 and controls the control-data-f or-synthesis read address 
generator 42 and the image-data-f or-synthesis read address 
generator 43. 

The control-data-f or-synthesis analyzer 46 controls 
"the repetition count 3117 of current image-f or-synthesis" 
included in the information of the header G2C read from the 
control-data-f or-synthesis storage 31 in the data-f or-synthesis 
storage 3, through the input terminal G2Ci n of the synthesis 
controller 4a. 

Suppose that the frame rate of the moving- image data 
GIG output from the image pickup device 1 is N*M (N and M are 
positive integers), as calculated by the frame rate detector 45. 
For example, suppose that N is 5, M is 3, and the frame rate is 
15 fps (frames per second). Here, "the repetition count 3117 
of current image-f or-synthesis " , which is included in the 
parameters of the header G2C stored in the control-data-f or- 
synthesis storage 31 within the data-f or-synthesis storage 3, 
is set to M (= 3) beforehand. This value is prepared so that 
the motion (frame rate) of the composite moving- image is 
optimized (becomes effective) at a frame rate of N*M. Here, 
the optimum and effective motion of the composite image is at 5 
fps . 

Then, suppose that the operating environment changes. 
For example, suppose that the ambient light becomes low and the 
frame rate of the moving- image data GIG output from the image 
pickup device 1 is multiplied by L/M (L is a positive integer) 
to be set to N*L . When L = 1, for example, the frame rate 
after the change is 5 fps. The frame rate detector 45 provides 
a calculation result of N*L and outputs the information to the 
control-data-f or-synthesis analyzer 46. When receiving the 
information of a frame rate of N*L, the control-data-f or- 
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synthesis analyzer 46 actually multiplies the repetition count 
by L/M and changes the repetition count to L. 

Because L = 1, the repetition count is 1 here. The 
actual rate of the image-data-f or-synthesis G2G is 5 fps, and 
the rate of the original image is kept at the optimum level 
(effective level). In other words, "the repetition count 3117 
of current image-f or-synthesis" included in the header G2C is 
set in correspondence with the reference frame rate (N*M in the 
example given above) . If the frame rate of the moving- image 
data GIG changes, the control-data-f or-synthesis analyzer 46 
automatically calculates in accordance with the frame rate 
information from the frame rate detector 45, and the control 
data read address generator 43 and the image-data-f or-synthesis 
read address generator 43 are controlled accordingly. 

As has been described above, the moving- image synthesis 
device of the third embodiment has the frame rate detector 45 
in the synthesis controller 4a and automatically changes the 
actual composite moving-image repetition count in accordance 
with the calculated frame rate, so that the optimum and 
effective motion of the original image can be maintained, and 
consequently, an optimum and efficient composite moving-image 
can be obtained. 

Further, the third embodiment has been described above, 
supposing that the frame rate of the moving- image data GIG 
output from the taken image 1 changes automatically. The frame 
rate can be changed intentionally as controlled by the CPU or 
the like. In that case, the frame rate need not be calculated 
by the frame rate detector 45. Accordingly, the same effect 
can be obtained by providing a current frame rate setting 
device instead of the frame rate detector 45. 

Furthermore, the frame rate detector 45 of the third 
embodiment detects the frame rate of the moving- image data 
output from the image pickup device 1 or another moving- image 
input stage. Because the synthesis controller 4 controls the 
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reading of the image-data-f or-synthesis G2G from the data-for- 
synthesis storage 3 in accordance with the frame rate detected 
by the frame rate detector 45, the motion of the image-data- 
f or-synthesis G2G stored in the data-f or-synthesis storage 3 
can be kept in the optimum and effective state even if the 
frame rate of the moving-image data changes. 

To be more specific , the synthesis controller 4a of the 
third embodiment controls the reading of the image-data-f or- 
synthesis G2G to the data-f or-synthesis storage 3 in such a 
manner that the repetition count of the image-data-f or- 
synthesis G2G becomes M when the frame rate of the moving-image 
data output from the image pickup device 1 or another moving- 
image input stage is N*M, and the repetition count of the 
image-data-f or-synthesis G2G from the data-f or-synthesis 
storage 3 becomes L when the frame rate becomes N*L . According 
to the third embodiment, the motion of the image-data-f or- 
synthesis G2G stored in the data-f or-synthesis storage 3 is 
kept in the optimum and effective state, or the state of frame 
rate N, even if the frame rate of the moving- image data changes. 

Except for the above described respects, the third 
embodiment is the same as the first embodiment. Further, the 
synthesis controller 4a of the embodiment can be applied to the 
moving- image synthesis device of the second embodiment. 

Modifications of the Present Invention 

The structure and the parameters of the control-data- 
f or-synthesis (header) G2C of the embodiments of the present 
invention are not limited to those shown in Fig. 4. The 
similar effect can be obtained even when the control-data-f or- 
synthesis (header) G2C has "a horizontal starting position of 
composite image", "a vertical starting position of composite 
image", "a horizontal ending position of composite image", and 
"a vertical ending position of composite image", instead of 
"the horizontal position 3111 of image-f or-synthesis " , "the 
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vertical position 3112 of image-f or-synthesis" , "the horizontal 
size 3113 of image-f or-synthesis" , and "the vertical size 3114 
of image-f or-synthesis" , or when the order of the parameters is 
changed as desired. 

Further, in the embodiments described above, there is 
no need to store the headers G2C continuously in the control- 
data-f or-synthesis storage 31 as shown in the figure or to 
store the items of image-data-f or-synthesis G2G continuously in 
the image-data-f or-synthesis storage 32. 

Furthermore, in the first embodiment and the third 
embodiment, the control-data-f or-synthesis storage 31 and the 
image-data-for-synthesis storage 32 are separately provided in 
the data-f or-synthesis storage 3. However, the control-data- 
f or-synthesis (header) G2C and the image-data-for-synthesis G2G 
may be stored in a united state or in a mixed state. 

Moreover, in the embodiments described above, when a 
sequence is formed, there is no need to match the order of the 
items of control-data-f or-synthesis (header) G2C stored in the 
control-data-f or-synthesis storage 31 with the order of the 
items of control-data-f or-synthesis (header) G2C forming the 
sequence. Similarly, the items of image-data-for-synthesis G2G 
stored in the image-data-for-synthesis storage 32 may be stored 
in any order . 
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